|
Author
|
Topic: Zelicoff 2007 rebuttal
|
rnelson
Member
|
posted 01-25-2008 02:47 PM
  
I've replicated Zelicoff's Monte Carlo simulation based on Honts 2002, and am contemplating a rebuttal at anti.Zelicoff is eager to include INCs as errors. I believe he is correct to include INCs in sensitivity and specificity estimates, but incorrecto to include them as errors in Bayesian simulation. Two main arguments stand out: 1) the simulation cannot represent what actually occurs in field practice, and is not reflected in administrative policies; and 2) his basic model is overly simplistic and does not adequately represent a bimodal polygraph testing context, in which some INCs are inevitable. Zelicoff's reported confidence intervals look really wide to me. I believe he may have incorrectly reported the two-standard-deviation range as the 95% CI. Does anyone know if that is correct in an MC simulation? I do not believe it is, and I believe the correct estimate is stil Mean +/- 1.96*(StDev/sqrt(of N)), for which I would use my sample-space of N=100 as a conservative estimate, even though the simulation includes 10,000 iterations of the sample space.
Here is a link to the MC replication
http://www.raymondnelson.us/other/Zelicoff_replication_1_1-24-08.xls
and some graphs. 
r ------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964) [This message has been edited by rnelson (edited 01-25-2008).] IP: Logged |
blalock
Member
|
posted 01-25-2008 03:20 PM
Zelicoff certainly needs rebutting. A "flip of a coin" (55 - 60%) average is misleading for sure. I don't even come close to understanding the statistics babble, but I have great disdain for the fact that he is knowingly manipulating the numbers to make a false declaration.------------------
Ben blalockben@hotmail.com IP: Logged |
Barry C
Member
|
posted 01-25-2008 03:42 PM
Ray,This just reminded me of an email I have overlooked. In any event, I don't know of any reason why that calculation for the CI would be wrong. I did some research and couldn't find anything to change my mind, but I'm no statistician. Maybe Dr. Lou, the stats professor, could weigh in. Have you considered running things by Stu Senter too? IP: Logged |
Ted Todd
Member
|
posted 01-26-2008 12:27 AM
Ray, Stat, Nelson and Barry,I will stipulate that you guys are all extremely intellegent. I am not. I keep trying to keep up with what is going on on this site but frequently feel like "dit dee dee"! Can your wisdom be put in a format that us regular guys can grasp? I love the conversation but feel like I am falling behind! Ted IP: Logged |
Bill2E
Member
|
posted 01-26-2008 06:15 AM
Just remember that figures don't lie, but liars can figure.Heard that years ago and it sure holds true, I am no statistician, my experience over 25 years is not a 50/50 chance, it is closer to Don's figures of 86% to 90% including inconclusive test results. To many that may sound low, I don't think it does. IP: Logged |
rnelson
Member
|
posted 01-26-2008 09:15 AM
[quote]Just remember that figures don't lie, but liars can figure.[/quite]Bill them' almost sounds like finghtin' words.
For anyone who's not interested in the propeller-headed math... Think of it as a form of Lunesta, and free and safer. And for those who really can't sleep, you can always read Grout's History of Western Music.
http://www.amazon.com/History-Western-Music-Sixth/dp/0393975274 - arguable the most boring book ever written (I'm fairly sure they put choloroform in the ink; just open it an snooze). ----------- Just knowing that answers exist makes us all smarter. The thing to keep in mind is that our techniques can still improve, but that doesn't happen by simply making up new thing and studying them in the same old ways (percent correct) - it takes getting up close and a lot more personal with the data. Here are some more graphs 
Whate we've got is decision accuracy at about 89% and INC at about 24%, using the blind scored higher quality studes reported by Honts 2002. This is all not strangely consistent with the findings of NRC/NAS 2003. OK, so how do you improve on this. First, descrease inconclusives. Second (not shown in this data), we must increase interrater reliability (inter-scorer concisistency), as that will set the upper limit of potential accuracy. So, the silliness looking at each other's charts is invaluable, as we learn to disagree less as we learn to emphasize data-drive principles more. It doesn't matter if we forget this all instantly. The overall effect will be inevitable improvement in our profession. Interested people get inspired, and the lowest-common-denomenator is slowly raised. OK, I'm off to the gym for some time on the rat-cage. Peace,
r
------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)
IP: Logged |
stat
Member
|
posted 01-26-2008 10:18 AM
Ted, I too at this time am incapable of understanding precisely what the hell Ray is talking about.
Perhaps some vocabulary lessons would be helpful.I have always been a fan of "The Dummies Guide to..." books. (I hear that guy is a Zilloinairre who started and franchised that series.)
Either way, what I do understand about statistics is that they are, regardless of our comfort in discussing them, polygraph's surviving best friend. If you live in rural areas where you must find a deep water source for your water supply, you can spend tens of thousands of dollars guessing where to drill (costing $20-50 a foot to have the drillers punch through your land.) When the average attempts and costs of mere guesswork come to around 4-5 failed water-finding attempts, and approximately 280-300 feet from those empty attempts ---$5,600 to $14,000 for nothin [ouch], many people like to consider options. Incidentally, I have heard of drillers having upwards of 100 failed attempts at finding water for a private land owner (YOUCH!) If you hire a successful Dowser (water witch), they (1 of 2 particularly successful guys in Southern Indiana) average 1 failed attempt before success. Dowsing was used by the Sumarians of thousands of years ago and even Moses had his own Dowser which the Isrealites revered and depended upon during the Exodus (?), and they are used today by large civil engineer firms secretly when they are surveying land for industrial use. Germany and Russia are now the Dowsing capitals. Dowsing is a bit of an embarrassment as we have no real clue as to why the successful ones are able to find water with a damn stick or a brass rod. Most Dowsers seem to be poor at it, but there are some that are BANKABLE every time. Remember, geological water table surveys are useless in deep country, and suggesting that drillers use educated guesses------well, drillers do one thing, and that's drill deep holes. Water surveys are better used as a benchmark for knowing how deep to drill, not necessarily where to drill. Incidentally, water witches do not and cannot find crude oil. strange. I would think that if they were charlatons, they would move up to a better game ($). I dunno. Now, before you "Spock types" get uneasy about lil ol' me comparing polygraph with water witching, lets remember that we still do not know exactly why the hell polygraph "works"----or at least we cannot agree to the details. I would NEVER, repeat NEVER use such an analogy in public, but when we speak of statistics, Dowsing....or better yet, Accupuncture have high statistical probabilities of success, regardless of our ability to explain why they "work." So, until the polygraph profession (if ever) can benefit from scientific mapping of the brain, and a host of other revelational advances, statistics are our best foundational SCIENTIFIC proof that what we do, works. Stories, anecdotes, whatever, do not impress the Science types, as many will not be able to get past the fact that we still have to manipulate our subjects, guess as to their fitness for testing, and score their results with one eye closed and our tongues perched outward, like an artist in concentration. my opinion only.
[This message has been edited by stat (edited 01-26-2008).] IP: Logged |
Barry C
Member
|
posted 01-26-2008 11:11 AM
Moses didn't have a dowser, and statistical analyses of dowsing data show they are no better at finding water than chance - unless you read the writings of dowsers. That, by the way, is a reason many are skeptical of studies published by those of us in the polygraph community, and it's one of the reasons I find these studies done by polygraph examiners in which they achieve 100% accuracy a little suspicious.IP: Logged |
stat
Member
|
posted 01-26-2008 11:42 AM
I am not attempting to rationalize Dowsing, although readers should know that in Exodus and Numbers, God allowed Moses and Aaron to use their staff to smite a rock and find water, as the wandering Isralites were gravely thirsty. In Theological circles, the biblical passages regarding water divination are very much argued. The point of the discussion was not to either impune or falsy attribute Dowsing to the bible, but instead was to reflect a sense of the idea regarding accomplishing something through unexplainable measures, be it from God, or the paranormal, or just dumb luck.
The greater point (which I am doing a piss poor job of making)is that polygraph, without statistics, finds itself as being a mysterious entity----and is a process that guessers will defer to for a better than guess probability. All the while, without statistical analysis, users will be left scratching their heads as to why the precise nature of the test "works" rather than being comfortable with "it just works."I will never again use a biblical concept alongside an analogy, as it cleary has too much baggage. Sorry for offending any readers. I myself do not believe in Dowsing, but I have hired one as an attempt to gain ANY statistical advantage over costly guesswork. 
[This message has been edited by stat (edited 01-26-2008).] IP: Logged |
Barry C
Member
|
posted 01-26-2008 11:57 AM
As a theologian, I think such stories are loaded with teaching. We just have to get them right. Moses didn't use his staff to locate water; God told him to speak to the rock (an act of faith) and water would come forth. Moses later hit the rock with his staff - which was contrary to what God told him to do, and he paid the price, albeit after getting water just the same. In any event, he wasn't dowsing in the normally understood sense of the word.For an interesting video see James Randi's website:
http://www.randi.org/joom/content/view/124/2/ Without good studies and statistics, we are little more than the James Van Praagh's of the world. So, I agree with your premise, but there is a difference between criterion validity (essentially, does it work?) and construct validity (the "why does it work?" part of the equation). IP: Logged |
Barry C
Member
|
posted 01-26-2008 12:01 PM
I should mention, James is a con man. A magician in essence tells you he'd going to fool you. It's an understood part of the game. James says what he does is real, and he's messing with people's lives - much like the "Moment of "Truth" (or bull).IP: Logged |
stat
Member
|
posted 01-26-2008 12:20 PM
Ah James. There is another figure we could discuss. A fraud for sure, but with a high degree of statistical success. Why? Like most "psychics", he is dislexic, a condition that rendered him as a child very dependant on his intuition, and highly hyper vigileant---the way a blind person can develope acute hearing for compensatiative purposes, James notices every detail of a person he views---their jewelry, their skin irregularities, their micro-facial expressions. He would make an extremely good interrogater with such senses----had he decided not to claim to hear ghosts and all of the other HS. Agreed, I misused the biblical analogy. Sorry.I am calling upon Ray to explain statistical analysis in a more heuristic way---using stories and Socratic method of parables so that we layman can appreciate the work that goes into the calculations, and also to appreciate how computers have replaced thousands of manhours in the math work. Also, I would like Ray to explain why in the below picture, he is wearing what appears to be either a swim-team whistle or a special olympic medal around his neck. Ray? 
For fun; Stare at the picture for a few moments and try to find the Giraffe;
 [This message has been edited by stat (edited 01-26-2008).] IP: Logged |
rnelson
Member
|
posted 01-26-2008 12:46 PM
special olympic medal.------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)
IP: Logged |
Barry C
Member
|
posted 01-26-2008 12:52 PM
Actually, when one studies cold reading - which is what honest magicians call it - statistics does come in to play - not great powers of observation. You simply ask about issues that give you a high probability of success, and yes, you could use that in an interrogation - and I have!IP: Logged |
Bill2E
Member
|
posted 01-27-2008 04:50 AM
Ray, Not fighting words at all, they were intended to exemplify the figures presented by the Anti site that our accuracy is 50%, or a coin toss. I have no problem with your math, and see by the graphs presented where we are going with this. I am. as stat has stated, simple minded and don't work well with formulas and equations, I conduct polygraph examinations based on research and training received as a result of those doing the research.
I do pay close attention to all of the research, not only that that supports the training I have received and the manner in which I conduct examinations. IP: Logged | |
Polygraph Place Bulletin Board
